{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "34ffbf85",
   "metadata": {},
   "source": [
    "## Using Evaluator-Optimizer Pattern to Generate and Evaluate Prospective Templates for AI Consulting Side Hustle"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c0454fae",
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "import json\n",
    "from dotenv import load_dotenv\n",
    "from openai import OpenAI\n",
    "from anthropic import Anthropic\n",
    "from IPython.display import Markdown, display\n",
    "\n",
    "load_dotenv(override=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9f00e59a",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Print the key prefixes to help with any debugging\n",
    "\n",
    "openai_api_key = os.getenv('OPENAI_API_KEY')\n",
    "anthropic_api_key = os.getenv('ANTHROPIC_API_KEY')\n",
    "google_api_key = os.getenv('GOOGLE_API_KEY')\n",
    "deepseek_api_key = os.getenv('DEEPSEEK_API_KEY')\n",
    "groq_api_key = os.getenv('GROQ_API_KEY')\n",
    "\n",
    "if openai_api_key:\n",
    "    print(f\"OpenAI API Key exists and begins {openai_api_key[:8]}\")\n",
    "else:\n",
    "    print(\"OpenAI API Key not set\")\n",
    "    \n",
    "if anthropic_api_key:\n",
    "    print(f\"Anthropic API Key exists and begins {anthropic_api_key[:7]}\")\n",
    "else:\n",
    "    print(\"Anthropic API Key not set (and this is optional)\")\n",
    "\n",
    "if google_api_key:\n",
    "    print(f\"Google API Key exists and begins {google_api_key[:2]}\")\n",
    "else:\n",
    "    print(\"Google API Key not set (and this is optional)\")\n",
    "\n",
    "if deepseek_api_key:\n",
    "    print(f\"DeepSeek API Key exists and begins {deepseek_api_key[:3]}\")\n",
    "else:\n",
    "    print(\"DeepSeek API Key not set (and this is optional)\")\n",
    "\n",
    "if groq_api_key:\n",
    "    print(f\"Groq API Key exists and begins {groq_api_key[:4]}\")\n",
    "else:\n",
    "    print(\"Groq API Key not set (and this is optional)\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3043cbc1",
   "metadata": {},
   "outputs": [],
   "source": [
    "prompt = \"\"\"\n",
    "I am an AI engineer living in the DMV area and I want to start a side hustle providing AI adoption consulting services to small, family-owned businesses that have not yet incorporated AI into their operations. Create a comprehensive, reusable template that I can use for each prospective business. The template should guide me through:\n",
    "\n",
    "- Identifying business processes or pain points where AI could add value\n",
    "- Assessing the business’s readiness for AI adoption\n",
    "- Recommending suitable AI solutions tailored to their needs and resources\n",
    "- Outlining a step-by-step implementation plan\n",
    "- Estimating expected benefits, costs, and timelines\n",
    "- Addressing common concerns or objections (e.g., cost, complexity, data privacy)\n",
    "- Suggesting next steps for engagement\n",
    "\n",
    "Format the output so that it’s easy to use and adapt for different types of small businesses.\n",
    "\"\"\"\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "77dcf06d",
   "metadata": {},
   "outputs": [],
   "source": [
    "print(prompt)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a02bcbc0",
   "metadata": {},
   "outputs": [],
   "source": [
    "competitors = []\n",
    "answers = []\n",
    "messages = [{\"role\": \"user\", \"content\": prompt}]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8659e0c3",
   "metadata": {},
   "outputs": [],
   "source": [
    "# First model: OpenAI 4o-mini\n",
    "\n",
    "model_name = \"gpt-4o-mini\"\n",
    "\n",
    "openai = OpenAI()\n",
    "\n",
    "response = openai.chat.completions.create(\n",
    "    model = model_name,\n",
    "    messages = messages\n",
    ")\n",
    "\n",
    "answer = response.choices[0].message.content\n",
    "\n",
    "display(Markdown(answer))\n",
    "competitors.append(model_name)\n",
    "answers.append(answer)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c27adf8d",
   "metadata": {},
   "outputs": [],
   "source": [
    "#2: Anthropic. Anthropic has a slightly different API, and Max Tokens is required\n",
    "\n",
    "model_name = \"claude-3-7-sonnet-latest\"\n",
    "\n",
    "claude = Anthropic()\n",
    "response = claude.messages.create(model=model_name, messages=messages, max_tokens=2000)\n",
    "answer = response.content[0].text\n",
    "\n",
    "display(Markdown(answer))\n",
    "competitors.append(model_name)\n",
    "answers.append(answer)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9ee149f9",
   "metadata": {},
   "outputs": [],
   "source": [
    "#3: Gemini\n",
    "\n",
    "gemini = OpenAI(api_key=google_api_key, base_url=\"https://generativelanguage.googleapis.com/v1beta/openai/\")\n",
    "model_name = \"gemini-2.0-flash\"\n",
    "\n",
    "response = gemini.chat.completions.create(model=model_name, messages=messages)\n",
    "answer = response.choices[0].message.content\n",
    "\n",
    "display(Markdown(answer))\n",
    "competitors.append(model_name)\n",
    "answers.append(answer)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "254dd109",
   "metadata": {},
   "outputs": [],
   "source": [
    "#4: DeepSeek\n",
    "deepseek = OpenAI(api_key=deepseek_api_key, base_url=\"https://api.deepseek.com/v1\")\n",
    "model_name = \"deepseek-chat\"\n",
    "\n",
    "response = deepseek.chat.completions.create(model=model_name, messages=messages)\n",
    "answer = response.choices[0].message.content\n",
    "\n",
    "display(Markdown(answer))\n",
    "competitors.append(model_name)\n",
    "answers.append(answer)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "63180f89",
   "metadata": {},
   "outputs": [],
   "source": [
    "#5: groq\n",
    "groq = OpenAI(api_key=groq_api_key, base_url=\"https://api.groq.com/openai/v1\")\n",
    "model_name = \"llama-3.3-70b-versatile\"\n",
    "\n",
    "response = groq.chat.completions.create(model=model_name, messages=messages)\n",
    "answer = response.choices[0].message.content\n",
    "\n",
    "display(Markdown(answer))\n",
    "competitors.append(model_name)\n",
    "answers.append(answer)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a753defe",
   "metadata": {},
   "outputs": [],
   "source": [
    "#6: Ollama\n",
    "ollama = OpenAI(base_url='http://localhost:11434/v1', api_key='ollama')\n",
    "model_name = \"llama3.2\"\n",
    "\n",
    "response = ollama.chat.completions.create(model=model_name, messages=messages)\n",
    "answer = response.choices[0].message.content\n",
    "\n",
    "display(Markdown(answer))\n",
    "competitors.append(model_name)\n",
    "answers.append(answer)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a35c7b29",
   "metadata": {},
   "outputs": [],
   "source": [
    "# So where are we?\n",
    "\n",
    "print(competitors)\n",
    "print(answers)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "97eac66e",
   "metadata": {},
   "outputs": [],
   "source": [
    "# It's nice to know how to use \"zip\"\n",
    "for competitor, answer in zip(competitors, answers):\n",
    "    print(f\"Competitor: {competitor}\\n\\n{answer}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "536c1457",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Let's bring this together \n",
    "\n",
    "together = \"\"\n",
    "for index, answer in enumerate(answers):\n",
    "    together += f\"# Response from competitor {index+1}\\n\\n\"\n",
    "    together += answer + \"\\n\\n\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "61600364",
   "metadata": {},
   "outputs": [],
   "source": [
    "print(together)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "be230cf7",
   "metadata": {},
   "source": [
    "## Judgement Time"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "03d90875",
   "metadata": {},
   "outputs": [],
   "source": [
    "judge = f\"\"\"You are judging a competition between {len(competitors)} competitors.\n",
    "Each model has been given this question:\n",
    "\n",
    "{prompt}\n",
    "\n",
    "Your job is to evaluate each response for clarity and strength of argument, and rank them in order of best to worst.\n",
    "Respond with JSON, and only JSON, with the following format:\n",
    "{{\"results\": [\"best competitor number\", \"second best competitor number\", \"third best competitor number\", ...]}}\n",
    "\n",
    "Here are the responses from each competitor:\n",
    "\n",
    "{together}\n",
    "\n",
    "Now respond with the JSON with the ranked order of the competitors, nothing else. Do not include markdown formatting or code blocks.\"\"\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d9a1775d",
   "metadata": {},
   "outputs": [],
   "source": [
    "judge_messages = [{\"role\": \"user\", \"content\": judge}]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c098b450",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Judgement time!\n",
    "\n",
    "response = openai.chat.completions.create(\n",
    "    model=\"o3-mini\",\n",
    "    messages=judge_messages,\n",
    ")\n",
    "results = response.choices[0].message.content\n",
    "print(results)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e53bf3e2",
   "metadata": {},
   "outputs": [],
   "source": [
    "results_dict = json.loads(results)\n",
    "ranks = results_dict[\"results\"]\n",
    "for index, result in enumerate(ranks):\n",
    "    competitor = competitors[int(result)-1]\n",
    "    print(f\"Rank {index+1}: {competitor}\")"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": ".venv",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.12.10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
